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The MAILING DATE of this communication appears on the cover sheet with the correspondence address - 
Period for Reply 

A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH(S) FROM 
THE MAILING DATE OF THIS COMMUNICATION. 

- Extensions of time may be available under the provisions of 37 CFR 1 .136(a). In no event, however, may a reply be timely filed 
after SIX (6) MONTHS from the mailing date of this communication. 

- If the period for reply specified above is less than thirty (30) days, a reply within the statutory minimum of thirty (30) days will be considered timely. 

- If NO period for reply is specified above, the maximum statutory period will apply and will expire SIX (6) MONTHS from the mailing date of this communication. 

- Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 U.S.C. § 1 33). 

- Any reply received by the Office later than three months after the mailing date of this communication, even if timely filed, may reduce any 
earned patent term adjustment. See 37 CFR 1.704(b). 

Status 

1 )S Responsive to communication(s) filed on 15 June 2001 . 
2a)D This action is FINAL. 2b)S This action is non-final. 

3) D Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 

closed in accordance with the practice under Ex parte Quay/e, 1935 CD. 1 1 , 453 O.G. 213. 

Disposition of Claims 

4) ^ Claim(s) 1-30 is/are pending in the application. 

4a) Of the above claim(s) is/are withdrawn from consideration. 

5) D Claim(s) is/are allowed. 

6) ^ Claim(s) 4 is/are rejected. 

7) S Claim(s) 1-3 and 5-30 is/are objected to. 

8) D Claim(s) are subject to restriction and/or election requirement. 

Application Papers 

9) E3 The specification is objected to by the Examiner. 

10)^ The drawing(s) filed on 15 June 2001 is/are: a)D accepted or b)^ objected to by the Examiner. 

Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1 .85(a). 

Replacement drawing sheet(s) including the correction is required if the drawing(s) is objected to. See 37 CFR 1 .121(d). 
1 1 )□ The oath or declaration is objected to by the Examiner. Note the attached Office Action or form PTO-1 52. 
Priority under 35 U.S.C. §§ 119 and 120 

12) D Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 1 19(a)-(d) or (f). 

a)DAII b)D Some*c)D None of: 

1 .□ Certified copies of the priority documents have been received. 

2. Q Certified copies of the priority documents have been received in Application No. . 

3. D Copies of the certified copies of the priority documents have been received in this National Stage 

application from the International Bureau (PCT Rule 17.2(a)). 
* See the attached detailed Office action for a list of the certified copies not received. 

13) D Acknowledgment is made of a claim for domestic priority under 35 U.S.C. § 1 19(e) (to a provisional application) 

since a specific reference was included in the first sentence of the specification or in an Application Data Sheet. 
37 CFR 1.78. 

a) □ The translation of the foreign language provisional application has been received. 

14) Q Acknowledgment is made of a claim for domestic priority under 35 U.S.C. §§ 120 and/or 121 since a specific 

reference was included in the first sentence of the specification or in an Application Data Sheet. 37 CFR 1 .78. 



Attach ment(s) 

1) |3 Notice of References Cited (PTO-892) 

2) O Notice of Draftsperson's Patent Drawing Review (PTO-948) 

3) (Zl Information Disclosure Statement(s) (PTO-1 449) Paper No(s) _ 



4) O Interview Summary (PTO-413) Paper No(s). 

5) □ Notice of Informal Patent Application (PTO-152) 

6) □ Other: 
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DETAILED ACTION 

Information Disclosure Statement 

1. The listing of references in the specification (i.e., 2 nd paragraph page 13) is not a 
proper information disclosure statement. 37 CFR 1 .98(b) requires a list of all patents, 
publications, or other information submitted for consideration by the Office, and MPEP 
§ 609 A(1 ) states, "the list may not be incorporated into the specification but must be 
submitted in a separate paper." Therefore, unless the references have been cited by 
the examiner on form PTO-892, they have not been considered. 

Drawings 

2. The drawings are objected to as failing to comply with 37 CFR 1 .84(p)(5) 
because they include the following reference sign(s) not mentioned in the description: 

- Item 20 (in figure 1 ) is not in the description. 

- Item 38 (in figure 2) is not in the description. 

3. The drawings are objected to because: 

- The meaning of construct 12 (in figure 1) is not clear or recognized as accepted 
practice as to the meaning of multiple unattached direction indicators. 

- The connection between 42->34 needs to be more clearly illustrated. 
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4. The drawings are objected to under 37 CFR 1 .83(a) because they fail to show 
"the flow changes" as described in the specification (first two lines of page 9). 

Any structural detail that is essential for a proper understanding of the disclosed 
invention should be shown in the drawing. MPEP § 608.02(d). 

The Examiner is proceeding with the understanding that either of the following 
remedies will be applied: 

- A "decision block" will be inserted between 32->40 indicating one invention 
operating selectively to route audio to recognizers. 

- Separate figures will depict 32->40->42->34 and 32->34 indicating unconditional 
routing to recognizers and not, respectively. 

5. A proposed drawing correction, corrected drawings, or amendment to the 
specification to add the reference sign(s) in the description, are required in reply to the 
Office action to avoid abandonment of the application. The objection to the drawings will 
not be held in abeyance. 

Specification 

6. The disclosure is objected to because of the following informalities: 

- The period is missing following the word "use" (7 th line from the end of page 4). 

- For consistency, the letter "r?" should be lower case (last 2 lines in page 3). 
Appropriate correction is required. 
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7. The title of the invention is not descriptive. A new title is required that is clearly 
indicative of the invention to which the claims are directed. 
The following title is suggested: 

"Selecting One of Multiple Speech Recognizers in a System on the Basis of 
Expectation of Performance Resulting From Experience". 



8. Claims 5 and 10 are objected to because of the following informalities: 

- In claim 5, the word "comprises" should be "comprised" (2 nd line). 

- Claims 10 and 16 are subject to interpretation. The term "recognizer information" 
could be interpreted to mean either Information about the recognizer 31 or 
"information from the recognizer 3 '. 

The Examiner is proceeding with the latter interpretation, as there were no 
enabling details disclosed in the Application (1 st paragraph page 9) that would 
indicate that associated recognizer information was from other than the provided 
results. 

Appropriate correction is required. 



Claim Objections 
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Claim Rejections - 35 USC § 103 

9. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

1 0. This application currently names joint inventors. In considering patentability of 
the claims under 35 U.S.C. 103(a), the examiner presumes that the subject matter of 
the various claims was commonly owned at the time any inventions covered therein 
were made absent any evidence to the contrary. Applicant is advised of the obligation 
under 37 CFR 1 .56 to point out the inventor and invention dates of each claim that was 
not commonly owned at the time a later invention was made in order for the examiner to 
consider the applicability of 35 U.S.C. 103(c) and potential 35 U.S.C. 102(e), (f) or (g) 
prior art under 35 U.S.C. 103(a). 

Kenne et al & Hsu et al 

11. Claims 1-3, 5-7, 9-10, 13-16, 18-19, 21, 23-24 & 27-29 are rejected under 35 
U.S.C. 103(a) as being unpatentable over Kenne et al ("Hybrid Language Models and 
Spontaneous Legal Discourse", 4 th International Conference on Spoken Language, 
October 1996) in view of Hsu et al (U.S. Patent 5,677,991 A). 
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12. Regarding claims 1 , 13, 18, 23 and 27; Kenne et al in describing hybrid language 
models reads on the features of the claim for selecting a recognizer from a number of 
recognizers as follows: 

- Kenne et al (with the "transcripts" input on page 717, section 2, Data - in the right 
column) reads on the feature of a) receiving an input stream; 

- Kenne et al (with the "perplexity" of the 2 nd paragraph from end of left column to the 
top of the right column) reads on the feature of b) deriving selection information, 
wherein the selection information includes performance-related information; 

- Kenne et al (with the "switching" of section 4, Results - in the right column page 
718) reads on the feature of c) using the selection information to select results from 
at least one enabled recognizer; 

- Kenne et al does not mention applications. Hsu et al (with the invention for flexible 
speech recognition that selects isolated word speech or continuous speech 
recognizers) reads on the feature of d) returning the results to the application (61 in 
figure 2). It would have been obvious to a person of ordinary skill in the art of 
speech signal processing at the time of the invention to apply the method/teachings 
of Hsu et al to the device/method of Kenne et al . providing a structure to put the 
strategy for selecting between recognizers into practice. 

- With specific regard to claim 18, the intrusion avoidance of Hsu et al (51 in figure 2) 
corresponds to and performs the function of the predictor ol the claim. 



Application/Control Number: 09/882,563 Page 7 

Art Unit: 2654 

13. Regarding claim 2, the claim is set forth with the same limits as claim 1. 
Kenne et al (lines 8-1 1 of section 4, Results in right column page 718) teaches the 
feature that causes a recognizer to be selected that is different than the recognizer used 
in a previous interaction (by virtue of the fact that a switch is made only if the perplexity 
changes, which would occur only if the current interaction differs from previous 
perplexities). 

Kenne et al does not mention that the selection information is updated; Hsu et al 
(lines 57-60) teaches this by setting the reference score as a baseline that is necessary 
to provide further evaluation. It would have been obvious to a person of ordinary skill in 
the art of speech signal processing at the time of the invention to apply the method 
and/or teachings of Hsu et al to the device/method of Kenne et al as a conventional 
programming method of preserving prior readings by storing them elsewhere to avoid 
their being overlaid by new measurements using the same algorithm code register 
fields. 

14. Regarding claim 3, the claim is set forth with the same limits as claim 1 . 
Kenne et al (with the formula in the 7 th line of the left column page 718) discloses the 
feature of deriving enabling information, and using the enabling information to enable at 
least one selected recognizer to process the input stream (with the "switch" made in the 
14 th line of the right column page 718). 
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1 5. Regarding claims 5 f 6 and 24; the claims are set forth with the same limits as 
claims 3, 1 and 23, respectively. Kenne et al (in the 2 nd paragraph of section 4, Results 
- right column page 718) reads on the feature that the enabling information - and that 
consequently the performance-related information - comprises at least one type of 
information from the group comprises of: channel characteristics, device characteristics, 
user information, contextual information, dialog state, recognizer costs and performance 
history. 

16. Regarding claims 7, 14 and 28; the claims are set forth with the same limits as 
claims 1,13 and 27, respectively. Kenne et al (in the 1 st line of page 718) records the 
classes of people into separate tracks to distinguish lawyers from witnesses, thereby 
reading on the feature that deriving the selection information further comprises 
analyzing the input stream for channel characteristics. 

17. Regarding claims 9, 15, 19 and 29; the claims are set forth with the same limits 
as claims 1 , 13, 18 and 27 respectively. Kenne et al (with Table 2 and the last 
paragraph of the right column, both on page 717) distinguishes between Lawyers and 
Witnesses (Statements & Questions), thereby reading on the feature of receiving 
contextual information associated with the input stream. 

18. Regarding claims 10 and 16 as understood by the Examiner, the claims are set 
forth with the same limits as claims 1 and 13, respectively. Kenne et al (with the 
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"switching" of section 4, Results - in the right column page 718) reads on the feature of 
receiving recognizer information from the enabled recognizers to be used in the 
selection information. 

19. Regarding claim 21, the claim is set forth with the same limits as claim 18. 
Kenne et al (with the "switching" of section 4, Results - in the right column page 718) 
reads on the feature that the predictor is operable to select a recognizer based upon the 
converted stream. 

Kenne et ah Hsu et al & Waibel et al' 957 

20. Claim 8 is rejected under 35 U.S.C. 103(a) as being unpatentable over Kenne et 
al in view of Hsu et al (U.S. Patent 5,677,991 A) and further in view of Waibel et al' 957 
(U.S. Patent 5,71 2,957 A). 

21 . Regarding claim 8, the claim is set forth with the same limits as claim 1 . 
Kenne et al & Hsu et al do not mention separate input devices so would not distinguish 
on that basis. The Waibel et al 957 invention of a speech repair and correction method 
for speech recognition system provides for different inputs (23 & 24 in figure 1 ) and 
consequently, using this information, locates error in hypothesis with highest score from 
list, generates control signals and second list, combines two lists and replaces error with 
hypothesis with highest combined score. 
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This reads on the feature of analyzing the input stream for device characteristics, 
which would have made it obvious to a person of ordinary skill in the art of speech 
signal processing at the time of the invention to apply the method/teachings of Waibel et 
al 957 to the device/method of Kenne et al & Hsu et al so as to allow the decision to be 
influenced by recognition that some devices increase the confidence of recognition 
results done on the streams they produce. 

Kenne et al Hsu et al & Kundu 

22. Claims 11-12, 17, 25-26 and 30 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Kenne et al in view of Hsu et al and further in view of Kundu (U.S. 
Patent 5,924,066 A). 

23. Regarding claims 1 1 , 1 7, 25 and 30; the claims are set forth with the same limits 
as claims 1,13 and 23, respectively. Kenne et al & Hsu et al do not mention feedback. 
With the invention for classifying a speech signal, Kundu (column 9 lines 49-54) reads 
on the feature of receiving feedback and including the feedback in the selection 
information. 

It would have been obvious to a person of ordinary skill in the art of speech 
signal processing at the time of the invention to apply the method/teachings of Kundu to 
the device/method of Kenne et al & Hsu et al so as to simplify training by including 
learned lessons on the basis of experience rather than attempting to predict all 
anticipated requirements. 
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24. Regarding claims 12 and 26; the claims are set forth with the same limits as 
claims 11 and 23, respectively. Kenne et al & Hsu et al do not mention feedback. 
Kundu (column 9 lines 50-51 ) reads on the feature that feedback is received from one of 
the group comprised of: off-line analysis, user feedback, and feedback from the 
recognizer. 

It would have been obvious to a person of ordinary skill in the art of speech 
signal processing at the time of the invention to apply the method/teachings of Kundu to 
the device/method of Kenne et al & Hsu et al to adjust for variations in confidence levels 
corresponding to changes in the input stream over time and with use. 

Kenne et al. Hsu et al & Waibel et al 000 

25. Claims 20 and 22 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over Kenne et al in view of Hsu et al (U.S. Patent 5,677,991 A) and further in view of 
Waibel et al 000 (U.S. Patent 5,855,000 A). 

26. Regarding claim 20, the claim is set forth with the same limits as claim 18. 
Neither Kenne et al nor Hsu et al mention confidence levels. Waibel et al 000 (column 1 
lines 55-67) teach the feature that the recognizers are also operable to provide 
individual-result confidence levels to the predictor. Because the determination is to be 
made between recognizers, it would have made it obvious to a person of ordinary skill in 
the art of speech signal processing at the time of the invention to apply the 
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method/teachings of Waibel et al 000 to the device/method of Kenne et al & Hsu et al in 
order to select the product of the recognizer providing the highest level of confidence. 

27. Regarding claim 22, the claim is set forth with the same limits as claim 18. 
Both Kenne et al and Hsu et al select after recognition. Waibel et al 000 (108 in figure 3) 
teaches the feature to select a recognizer prior to the recognizer receiving the input 
stream (column 7 lines 57-60) which would have made it obvious to a person of ordinary 
skill in the art of speech signal processing at the time of the invention to apply the 
method/teachings of Waibel et a i 000 to the device/method of Kenne et al and Hsu et al 
because there is no need to operate recognition when controlling information would 
override selection. 



28. Claim 4 is objected to as being dependent upon a rejected base claim, but would 
be allowable if rewritten in independent form including all of the limitations of the base 
claim and any intervening claims. 



Allowable Subject Matter 



29. As allowable subject matter has been indicated, applicant's reply must either 
comply with all formal requirements or specifically traverse each requirement not 
complied with. See 37 CFR 1.111(b) and MPEP § 707.07(a). 
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30. The following is a statement of reasons for the indication of allowable subject 
matter: 

- The present invention is directed to processing speech using one of several 
available recognition devices. 

- Claim 4 identifies the uniquely distinct feature "to enable a recognizer based upon its 
expected future performance ". 

- The closest prior art of Kenne et al discloses that "the enabling information is used to 
enable a recognizer" but fails to anticipate or render the above underlined limitations 
obvious. 

Conclusion 

31 . The prior art made of record and not relied upon is considered pertinent to 
applicant's disclosure. 

- Araqon (U.S. Patent 6,055,327 A) detecting data entry errors by sorting amounts 
and verifying amount order. 

- Kundu (U.S. Patent 5,924,066 A) classifying a speech signal. 

- Barry et al ("The Simultaneous Use Of Three Machine Speech Recognition Systems 
To Increase Recognition Accuracy", Proceedings of the IEEE 1994 National 
Aerospace and Electronics Conference, May 1994) ASR in parallel combine speech 
recognition accuracy better than by each of the individual systems alone. 
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- Yashchin etal ("Performance Of Speech Recognition Devices: Evaluating Speech 
Produced Over The Telephone Network", International Conference on Acoustics, 
Speech, and Signal Processing, May 1989) performance of four ASR devices reflect 
the realistic conditions under which the digits were obtained. 

- Wang et al ("Towards Universal Speech Recognition", Fourth IEEE International 
Conference on Multimodal Interfaces, October 2002) teaches maintaining several 
monolingual engines. 

32. Any inquiry concerning this communication or earlier communications from the 
Examiner should be directed to Daniel A. Nolan at telephone (703) 305-1368 whose 
normal business hours are Mon, Tue, Thu & Fri, from 7 AM to 5 PM. 

If attempts to contact the examiner by telephone are unsuccessful, supervisor 
Richemond Dorvil can be reached at (703)305-9645. 

The fax phone number for Technology Center 2600 is (703)872-9314. Label 

informal and draft communications as "DRAFT" or "PROPOSED", & designate formal 

communications as "EXPEDITED PROCEDURE". Formal response to this action may 

be faxed according to the above instructions, 

or mailed to: P.O. Box 1450 

Alexandria, VA 22313-1450 

or hand-deliver to: Crystal Park 2, 

2121 Crystal Drive, Arlington, VA, 
Sixth Floor (Receptionist). 
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Any inquiry of a general nature or relating to the status of this application or 
proceeding should be directed to Technology Center 2600 Customer Service Office at 
telephone number (703) 306-0377. 

Daniel A. Nolan 
Examiner 
Art Unit 2654 

DAN/d 

December 22, 2003 — — 



DANIEL NOLAN 
PATENT EXAMINER 




